Revolutionizing Cancer Risk Prediction: The LFSPRO Validation Journey and its lmpact on Clinical Practice
Dr. Wenyi Wang
Dr. Wang is a Professor of Bioinformatics and Computational Biology and Biostatistics at the University of Texas MD Anderson Cancer Center. She received her PhD from Johns Hopkins University and performed postdoctoral training in statistical genomics and genome technology at UC Berkeley with Terry Speed and at Stanford with Ron Davis. Wenyi’s research includes significant contributions to statistical bioinformatics in cancer, including MuSE for subclonal mutation calling, DeMixT for transcriptome deconvolution, Famdenovo for de novo mutation identification, and more recently, a pan-cancer characterization of genetic intra-tumor heterogeneity in subclonal selection and in tumor cell transcriptional plasticity respectively. Her group is focused on the development and application of computational methods to study the evolution of the human genome as well as the cancer genome, and further develop risk prediction models to accelerate the translation of biological findings to clinical practice.
Dr. Hoai Nam Nguyen
Hoai Nam Nguyen defended his PhD thesis in Statistics on “Characterization of cancer development and recurrence through mathematical and statistical modeling” under the supervision of Dr. Wenyi Wang, Dr Marek Kimmel, at Rice University on March,22,2024.
Regarding the research background and significance, does this work discover new knowledge or solve existing problems within the field? Please elaborate in detail.
Our work, which was recently published in the Journal of Clinical Oncology (JCO), contributes to the resolution of an existing problem within the field: there is a plethora of risk predictions models that have been developed for hereditary cancer syndrome, but the fact remains that very few of them are being utilized in real clinical settings. We believe that a major obstacle is to convince healthcare providers (e.g., clinicians, genetic counselors) to use these data-driven approaches. Our software suite, LFSPRO, consists of two risk prediction models to characterize cancer risks in families with Li-Fraumeni syndrome (LFS). In order to overcome the mentioned hurdle, we conducted a validation study of LFSPRO on a patient cohort that was collected as part of the Clinical Cancer Genetics (CCG) program at MD Anderson Cancer Center (MDACC). Unlike most other cohorts, which are meticulously collected for research purposes, our unique patient cohort closely resembles what genetic counselors observe in their counseling sessions, where data collection only takes up 30 minutes of time. Our validation results were indeed lower compared to the previous validation studies on research cohorts, but still at a reasonable level. Due to the nature of our validation dataset, these results provided strong indication that LFSPRO had the potential to assist decision making in clinics. In a bigger picture, our paper shows that model performance can deteriorate on less well-behaved datasets, hence highlighting the importance of clinical validation studies in bridging the gap between methodology research labs and clinics. (Also see MDA news)
How did the reviewers evaluate (praise) it?
Our manuscript was evaluated by both clinical and statistical reviewers. It was also read by the editors during the peer review process. In general, they recognized the significance and uniqueness of our study, as well as the efforts that we made to help families with hereditary cancer syndromes via a statistically sound approach. Associate editor Robert G. Maki MD/PhD/FACP/FASCO from MSKCC wrote the relevance of our work: A novel model helps better predict risk of cancer development in patients with Li-Fraumeni syndrome. These types of models, along with primary cancer screening, will hopefully improve the care for patients with familial cancer syndromes. Quotes from the anonymous reviewers: “The validation of a risk prediction model for LFS that can be used clinically would change clinical practice for the better, I applaud the authors for their work in this field.” “The strengths of this work include its potential to provide useful modeling despite the ongoing problem of missing information in a single clinical genetic counseling session.”
If this achievement has potential applications, what are some specific applications it might have in a few years?
Our goal is to disseminate the risk prediction models we developed to clinics to assist decision making. Currently, we are experimenting them with a small group of genetic counselors at MDACC, and performing a concordance study to evaluate how LFSPRO performs relative to the more established clinical criteria (e.g., Chompret criteria) in predicting deleterious germline TP53 mutations, prospectively. More specifically, our publication is based on a retrospectively collected patient cohorts from the clinic. We are currently writing up new findings from a prospectively collected patient cohort seen by genetic counselors over the past two years at MD Anderson. As these results remain positive, our clinical collaborator has engaged a patient population to contemplate on a randomized clinical trial on LFSPRO, which should represent the last step before its full integration into clinics. We have actually received requests from several clinics in the US to use LFSPRO right away.
Can you recount the specific steps or stages from setting the research topic to the successful completion of the research?
Given the raw dataset, we preprocessed it to obtain a reformatted dataset that was ready for LFSPRO. This stage involved frequent communication between us and our clinical collaborators to understand the data in more details and to clarify ambiguity.
In a typical counseling session, which is about an hour in length, the genetic counselor spends the first 20-30 minutes to collect family history of the counselee before performing a comprehensive risk assessment. As such, the collected information only represents a snapshot of the family history, with a lot of family members having missing ages at last contact or missing ages at cancer diagnoses. Unlike research-based datasets, there are usually no time and resources to obtain more complete information. Thus, an important part of our validation study was to agree on a way to handle this issue of missing information.
Since the goal of our study was to expedite the clinical utility of LFSPRO, we also wanted our validation study design to closely resemble real clinical settings. For example, since the National Comprehensive Cancer Network (NCCN) guidelines recommend testing for all close relatives of a positively tested individual, we strongly emphasized the models’ utility to accurately predict TP53 mutations when no genotype information is available within the family. When selecting statistical measures of model performance, we decided to comprehensively evaluate the models in both discrimination and calibration, since both of them are important in clinical decision making.
Finally, we spent a lot of time writing up the manuscript. Even though all the analyses had been done, it was challenging to highlight the results in the best possible way. We, for example, constructed a figure to visually describe how the data collection process is different for research protocol-based and clinical counseling-based cohorts. This figure received positive comments from one of the clinical reviewers during the peer-review process. We then further computed key summary statistics to emphasize the differences between the two types of patient cohorts. We believe that these changes helped deliver the key points to the reviewers at JCO, who are not familiar with the statistical modeling or the technical language. Again, the clinical expertise of our collaborators was really helpful during this final stage.
Were there any memorable events during the research? You can tell a story about anything related to people, events, or objects.
Nam: Due to the clinical nature of our study, we closely worked with our clinical collaborators at MDACC at all stages during the research process. I started working on this project when I was still a junior PhD student in Statistics at Rice University. Coming from a technical background with a bachelor degree in Math and little collaboration experience, I soon realized that statisticians and clinicians had very different communication styles. In particular, I had difficulty communicating effectively with our collaborators and conveying my research ideas and results in the manuscript in a way that was approachable to the reviewers. Nevertheless, I recognized that I would need to adapt to this new setting in order to expedite the productivity of our research group as a whole. We went back and forth a lot, especially during the manuscript stage, to find a way that best emphasized the clinical utility of our risk prediction models. With the help of my PhD advisor Wenyi Wang, I am grateful that I managed to overcome this hurdle and push the project to the finish line.
Wenyi: When I received this dataset from a newly formed collaboration with Dr. Banu Arun, co-director of the Clinical Cancer Genetics program at MD Anderson, I was still feeling sad about losing my 10-year long collaborator Dr. Louise Strong (to retirement), who is considered “the mother of LFS” and spent 40 years collecting/updating meticulous data from families with LFS. We published many modeling and cancer research papers together. But she had to retire. I also did not like the significant amount of missing data in this new dataset, as a statistician data snob. But somehow, in the process of this study, through meeting with Banu’s GC team every week, a lightbulb went off. I realized that this dataset is much more closer to reality and we will have to show LFSPRO works in this data setting to change many clinicians and GCs’ mindset: mathematical models may work with perfect data, but clinical criteria consisting of simple sentences are still better for the clinical decision-making. This mindset has prevented the field of risk prediction modeling from moving forward.
Is there a follow-up plan based on this research? If so, please elaborate.
We are conducting a prospective concordance study on LFSPRO to further evaluate its clinical utility in predicting deleterious germline TP53 mutations. We hope to achieve better results than the Chompret criteria, which are recommended in the National Comprehensive Cancer Network (NCCN) guidelines. If the results support our models, it will be another major step towards our end goal.
We are also developing a joint statistical framework that estimates cancer-specific age-at-onset penetrances beyond the first primary cancer, thus having the potential to assist risk assessment for cancer survivors. This model can achieve prediction objectives that the current models in LFSPRO cannot. Our manuscript has recently been submitted to a statistical journal. We plan to integrate this new model to LFSPRO to have a complete software suite for risk predictions in families with Li-Fraumeni syndrome. In order to validate the clinical utility of the joint model, we will perform an extensive validation study on multiple patient cohorts from different cancer institutes across the United States and from Germany. This ensures that the model is robust to variations between datasets (e.g., data quality, ascertainment criteria).
Without a doubt, AI is one of the hot topics of 2023, requiring extensive data support in its development. What assistance can biostatistics offer to the development of AI?
AI offers a wide range of machine learning methods to automate the analysis of complex datasets in healthcare and biomedical science (e.g., medical imaging data were manually examined by radiologists). However, there are still a lot of data types that are more relevant to traditional statistical methods (e.g., time-to-event patient data), and we see no reasons that their significance will diminish in a foreseeable future. Analyses of these different data types are not necessarily separate, since there have been hybrid models that attempt to combine them to achieve better prediction results. This represents a collaboration opportunity between biostatisticians and AI developers. In addition, bridging the gap between AI research labs and clinics is highly challenging since most of these AI models are black boxes (i.e., we do not understand how the output is computed). Extensive validation studies are, therefore, even more relevant to ensure that the models can perform well in all real-world settings. Biostatisticians, with their expertise in clinical trials and study design, can help expedite this process.
Besides the above questions, is there anything else about this achievement that you would like to add? If so, please add it below.
We would like to thank the patients for contributing their data to our studies. The LFS community is a very special one that is dear to my heart. Every two years, the LFS association (https://www.lfsassociation.org/) holds a 2-day conference that brings clinicians, biologists (computational and experimental) and patients together. My team has been attending this conference over the past 10 years. Talking with these patients and their family members has been an extremely humbling and motivating experience. Although my laboratory research primarily focuses on tumor heterogeneity and evolution, I always find joy in spending time on this completely separately line of research.
Proofread by: Hongtu Zhu